How to (accurately) skip past streams

For processing massive data streams, most proposed algorithmic methods look at each new item, perform a small number of operations while keeping a small amount of memory, and still perform much-needed analyses. However, in many situations, the update speed per item is very critical and not every item can be extensively examined. In practice, this has been addressed by sampling only a subset of items (say 1 in N) from the input, but it results in loss of guarantees on the accuracy of the post-hoc analyses. In this paper, we present a technique of skipping past streams. Unlike traditional sampling approaches, our skipping is performed in a principled manner without significant loss of guarantees on post-hoc analyses, while substantially improving the processing rate. Using this technique on top of well-known sketches, we show improvements in the update time as well as guaranteed accuracy for a number of stream processing problems including data summarization, heavy hitters detection and self-join size estimation. We present experimental results of our methods over synthetic data and integrate our methods into Sprint’s Continuous Monitoring (CMON) system for live network traffic analyses. Furthermore, going beyond traditional packet header analyses, we show how the packet contents can be analyzed at streaming speeds, a more challenging task because each packet content can result in many updates.